was this some kind of a bug that was present before?

Yes, this was a latent pre-existing bug exposed by the migration.

Before this PR, was str(uuid7()) — a fresh random string on every run. Since the dedup hash included the record_id column and that value changed every invocation, the cached-source dedup never actually recognised repeat rows as duplicates. It was silently broken.

The migration made it visible: deterministic UUID v5 record_ids meant the hash would be stable across runs, which is correct — but we also don't need to include the record_id in the hash at all, since it's derived from the source_id + provenance token already captured in the hash. So the fix excludes record_id tag columns from the dedup hash (removing redundant information) while making the dedup work correctly for the first time.

how is this value chosen?

It was arbitrary — chosen once by the implementer with no principled derivation, which is why it looked opaque.

Fixed: replaced with a UUID v5 derived from uuid.NAMESPACE_URL + a stable OrcaPod-specific URL string, so the value is reproducible from its inputs rather than being a magic hex literal:

_SOURCE_RECORD_ID_NAMESPACE = uuid.uuid5(
    uuid.NAMESPACE_URL,
    "https://orcapod.io/namespaces/source-record-id",
)

Added a doc comment noting this value must never change once deployed, since changing it would invalidate all existing stored record IDs.

wouldn't this be cleaner to return an actual UUID object instead?

Agreed — done. datagram_id now returns uuid.UUID throughout the protocol, Datagram, and Tag/Data. Generation uses uuid7() normalised to stdlib uuid.UUID (for cross-library equality). Arrow array construction calls .bytes at the point of actual serialisation, keeping the Python interface ergonomic.

which constant would have triggered the invocation of pyarrow and why?

Accessing either constant triggered it. The __getattr__ in types.py called import pyarrow as _pa to construct the value on first access. So any from orcapod.types import UUID_ARROW_TYPE or from orcapod import UUID_ARROW_TYPE call would load the full pyarrow C-extension immediately — which made the __init__.py eager import of the constant just as non-lazy. The __init__.py __getattr__ was needed to defer even that.

This whole mechanism is removed now per the overall review direction: the constants are gone, both __getattr__ functions are removed, and pa.binary(16) is used inline at each site.

perform a more thorough test based on the target column

Done. The assertion now checks per column: record_id columns (matching SYSTEM_TAG_RECORD_ID_PREFIX) assert isinstance(val, bytes) and len(val) == 16; all other system tag columns assert isinstance(val, str). This catches any accidental type regression in either direction.

Is there a good reason to mix uuid_utils and stdlib for handling uuid? Does uuid_utils offer richer set of functions for uuid generation/handling?

Agreed — is only here for UUIDv7 generation (time-ordered, monotonic), which isn't in the stdlib before Python 3.12. However and are distinct types that don't compare equal even for identical bit patterns, so we normalise immediately to on creation. The comment has been expanded to make this explicit.

We should include tests to ensure that the removal force convresion into str is not creating a new error?

Done. Added in . It iterates over all system-tag columns in the predicted schema after a two-way join and asserts that record_id columns map to (from ) and source_id columns map to — verifying that is carried through unchanged.

I acutally don't understand the logic here. For system tag, the source_id is the unique specifier of the nature/identity of the source and the record_id is a unique specifier about the origin of the specific record. If we are deeming record_id to be truly redundant with record_hash, then we should use record_id as the record_hash. Shouldn't it be the case that we'd actually include everything that's part of the identity of the row as part of the record hash so that in case two entries with everything identical BUT the record_id, suggesting that they originated from two distinct entries in the source that happens to carry identical values, we can correctly distinguish them? Otherwise, of course the inclusion of the rest of the fields are important to detect situation where same record_id entries end up having distinct values but we still want to track it.

Agreed — I reverted the exclusion. The record_id system tag encodes row origin (source identity + provenance token), so two rows that are byte-identical in their data columns but originate from different positions in the source will have distinct record_id values and must be stored as separate entries. Hashing the full live_table (including the record_id column) is correct. The dedup logic now hashes all system tag columns.

What is this typically set to? Also be sure to use Orcapod instead of OrcaPod throughout the codebase. If this is meant to be a namespace URL that we would maintain, make it https://orcapod.org/namespaces/source-record-id

Fixed:

• URL corrected to https://orcapod.org/namespaces/source-record-id (was http://orcapod.ai/...)
• Capitalisation corrected to "Orcapod"
• Added the pre-computed value as a comment: uuid.UUID('877ec89b-6645-5852-ba37-a94604043f5e')

let's instead return uuid.UUID for consistency?

Fixed — _make_record_id_bytes renamed to _make_record_id and now returns a uuid.UUID rather than raw bytes. The call site does .bytes to get the 16-byte value for Arrow storage. This gives callers a real UUID object (with full UUID API surface) instead of an opaque byte string.

why do we still have to take in str?

Dropped str entirely. compute_pipeline_entry_id now returns bytes (a method-prefixed digest via ContentHash.to_prefixed_digest()), and UUID-based record IDs are stored as 16-byte raw values — both are bytes. There is no longer a caller path that passes a str record ID, so the str union arm was just dead code.

why is this necessary? What currently can yield incompatible str record_id?

Cleaned up — record_id is now bytes only and _ensure_record_id_column always creates a pa.large_binary() column (no conditional branch on type). pa.large_binary() is used uniformly for all database record ID columns: it accommodates both fixed-size UUID bytes (16 bytes) and variable-length method-prefixed pipeline entry digests without requiring a schema change if sizes shift.

we should drop support for large_string

Dropped large_string support entirely. _ensure_record_id_column now always creates a pa.large_binary() column unconditionally, and all str | bytes / str type annotations in the Delta Lake DB layer have been updated to bytes only — matching the protocol and the other database implementations.

What's the need to name this _uuid_module?

Fixed — removed the alias. Changed to plain import uuid and updated the usage to uuid.UUID(bytes=uuid7().bytes).

Fixed — changed add_system_tag_columns() to use pa.binary(16) for the record_id column. Record IDs in system tags are always UUID v5 bytes (exactly 16 bytes), so pa.binary(16) is the precise type. The docstring and the implementation now agree.

Fixed in commit e0bbc37. add_system_tag_columns() now uses pa.binary(16) for the record_id column — both the field declaration and the pa.array() call — matching the docstring.

Fixed — same change as above. add_system_tag_columns() now appends the record_id column as pa.binary(16) (both the field definition and the pa.array() call). The docstring was already correct; the implementation is now consistent with it.

Fixed in commit e0bbc37 (same change as above — both the field and the array call now use pa.binary(16)).

Fixed — reverted _pg_type_to_arrow("uuid") back to pa.binary(16), and updated _coerce_pg_value to extract .bytes from psycopg's uuid.UUID objects instead of calling str(). Unit test (test_uuid) and integration test (test_all_pg_types) updated accordingly.

Fixed in commit e0bbc37. _pg_type_to_arrow("uuid") now returns pa.binary(16).

Fixed in commit e0bbc37. _coerce_pg_value() now checks arrow_type == pa.binary(16) and returns value.bytes (raw 16 bytes) instead of str(value).

Fixed in commit e0bbc37. test_uuid now asserts pa.binary(16).

Fixed in commit e0bbc37. The integration test now asserts infos["i"].arrow_type == pa.binary(16) for the UUID column.

No code change needed — updated the PR description instead. Adding module-level constants to types.py would trigger PyArrow import at types import time, defeating the LazyModule pattern already used there. Each call site keeping its own pa.binary(16) literal is clear and avoids that footgun. The PR description table row for types.py now reflects this.

The code is correct — datagram_id intentionally returns uuid.UUID (not bytes), as agreed in a prior review round. The PR description's Breaking Change note was wrong. Updated it to correctly state: datagram_id → uuid.UUID; record_id/_log_id/_status_id → bytes (16 bytes).

Fixed — docstring now says pa.binary(16) instead of pa.large_binary().

Fixed — replaced f"{source_id}::{provenance_token}" with the length-prefixed form f"{len(source_id)}:{source_id}:{provenance_token}". Since the length of source_id is embedded, the parser can always locate the boundary exactly, even when either component contains : or ::. This is a pre-v0.1.0 hash change (intentional; no backward-compat obligation).

Fixed — both the pl.Series and pa.Array/pa.ChunkedArray branches now apply coerce_record_id element-wise, matching the Collection branch. All three paths now produce a list[bytes].